1. Field of the Invention
The present invention relates to the automatic measurement of visually perceptible attributes of a person or a group of people within the range of sensors and combining this information with the transaction records that are generated by the person or people executing a transaction in a retail store.
2. Background of the Invention
Retailers and consumer goods companies are constantly looking for new ways to improve their retail performance. Companies spend millions of dollars each year in an effort to understand the buying habits of their customers. Until now, businesses have had to rely on spot surveys and ineffective frequent buyer programs to try to segment the customers based on various attributes and understand their behaviors in their establishment.
One of the crucial attributes dictating customer behavior is their demographic profile. There is a very strong relationship between the demographic profile of the customers—gender, age, and ethnicity—and their shopping habits—what they buy. Each demographic group engages with products in the stores differently and therefore retailers and consumer goods companies must develop targeted strategies to reach and convert specific segments. On top of that, the shopping behavior also changes when the customers shop in groups, such as family members or friends. If this kind of data can be collected, it can be further analyzed to provide crucial information to retailers or marketers.
Since the purchase data—time, location, and items—of the customers are registered through electronic cash registers, the missing piece of information is the demographic profile of the customers. Customer loyalty programs, such as frequent shopper club cards and customer panel data, have been widely used to collect shopper demographics data. The data collected from loyalty programs—household data—is useful for analytics because it helps analysts link multiple transactions to the same loyalty card. This helps them analyze the repetitive purchasing behavior of the card users. Customers are expected to provide information regarding their demographics, income, household, etc. when they apply for a card. These details are used by the analysts to group the transaction data for different customer segments. Although the insights gathered from such analyses are extremely valuable, they also suffer from major drawbacks. Since customers are expected to actively participate in the data collection process, the household data has multiple biases. The household data is limited to the customers who use the loyalty card at the checkout; it does not include the customers who do not participate in the programs or who forgot to carry their cards. Often the demographics information provided at the time of registration is outdated, inaccurate, or incomplete, making it highly unreliable. Finally, the loyalty card data does not provide highly valuable information such as who is the primary shopper in the household, when customers shop alone and when they shop as a group, etc.
It is one of the main goals of the present invention to augment the depth and accuracy of existing data by addressing the shortcoming of household data by measuring the demographics information and the shopping group information without relying on any intrusive (such as interview), inefficient (passive human observation), or inconvenient/incomplete (customer loyalty card) means. The proposed method utilizes video cameras installed at the checkout counters to measure the demographics information and group information of the checkout customers. The method is also capable of measuring the checkout purchase behavior—whether items such as candy or magazines have been picked up at the checkout shelves, which is impossible to deal with using customer loyalty programs.
The present invention utilizes one or more video cameras to recognize the demographic profiles as well as the group information of the shoppers. At least one camera captures the video of facial images of the customers waiting at the checkout queue, and tracks the facial images individually to determine the demographic classes of each person. The same video can be analyzed to determine the group information—whether some of the customers in the queue are family members or friends shopping together. Further facial expression analysis of the people in the group can also determine which person is the “leader” (who takes the role of interacting with the cashier and makes the payment) of the group and estimate the shoppers' overall emotional response during the transaction. An optional top-down camera can also be installed and utilized to infer more accurate group information and to identify potential checkout purchases.
U.S. Pat. No. 5,369,571 of Metts disclosed a method and apparatus for obtaining demographic information at the point of sale. In the disclosed prior art, the sales clerk had to input an assessment concerning certain demographic information about the customer to generate the demographic data, using a bar code scanner, a keypad with keys, or buttons. However, this increases the sales clerk's labor load, and the extra activities delay the overall processes at the point of sale, which could be costly from the business point of view of the particular business facility. Furthermore, the assessment could vary depending on the personal viewpoints of the sales clerks, thus making the accumulated demographic data from different sales clerks over a period of time unreliable. These problems in the prior art require an automatic and efficient approach for gathering the demographic information from the customers.
More recently, there have been attempts to track customers in a retail setting. For example, in Ismail Haritaoglu, Myron Flickner, “Detection and Tracking of Shopping Groups in Stores,” cvpr, vol. 1, pp. 431, 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'01)—Volume 1, 2001, (hereinafter Haritaoglu 1), Haritaoglu 1 describes a technique for tracking groups of people as they move through the store.
Computer vision algorithms have been shown to be an effective means for detecting people. For example, in Ismail Haritaoglu, Myron Flickner, “Attentive Billboards,” iciap, pp. 0162, 11th International Conference on Image Analysis and Processing (ICIAP'01), 2001, (hereinafter Haritaoglu 2), Haritaoglu 2 describes a method for detecting people and determining how long those people looked at a billboard. Also, in U.S. Pat. Appl. Pub. No. 20020076100 of Luo Jiebo (hereinafter Luo), the author describes a method for detecting human figures in a digital image.
Other computer vision techniques have been shown to be able to extract relevant demographic features of people in an image. For example, in Moghaddam, Baback and Yang, Ming-Hsuan, “Gender Classification with Support Vector Machines”, 2000 Proc. of Int'l Conf. on Automatic Face and Gesture Recognition, the authors describe a technique using Support Vector Machines (SVM) for classifying a face image as a male or female person. In the U.S. Pat. No. 5,781,650 of Lobo, et al. (hereinafter Lobo), the authors describe a method for discerning the age of a person in an image. In Lyons, Michael J. et al, “Automatic Classification of Single Facial Images”, 1999 IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 12, pp. 1357-1362, the authors describe a method for discerning the ethnicity (or race) of a person in an image. Finally, in U.S. Pat. No. 6,188,777 of Darrell, et al. (hereinafter Darrell), Darrell describes a means to extract a person's height, skin color, and clothing color from an image of a person. The combination of computer vision techniques such as the ones mentioned above allows for the possibility of connecting the visual information from a scene with a timestamp and a location marker to derive rich behavioral characteristics of the people in the scene.
Combining these extracted features with transaction data records will provide a better understanding of the types of people who are transacting. Until now many ideas have been put forth in an attempt to understand the characteristics of the people who are actually transacting. For example, in the U.S. Pat. Appl. Pub. No. 20020052881 of Player, Zen (hereinafter Player), Player describes a method for collecting the demographics of people while they are playing a game on the Internet. In U.S. Pat. No. 6,070,147 of Harms, et al. (hereinafter Harms), Harms describes a customer loyalty program using a government issued identification card, like a driver's license. In U.S. Pat. Nos. 5,974,396, 6,298,348, 6,424,949 and 6,430,539, consumer profiling is performed based on a consumer's past shopping history. These techniques give rise to the privacy question, and/or they fall short when the consumer does not remember their loyalty card. U.S. Pat. No. 6,285,983 of Jenkins attempts to address the privacy issue while still marketing to the consumer based on their profile. U.S. Pat. Nos. 6,393,471, 6,408,278, 6,338,044, 5,983,069, 6,129,274, and U.S. Pat. Appl. Pub. Nos 20020035560 and 20010004733 all discuss the delivery of customized content and/or advertisements based on consumer profiles.